Skip to content

Add async support for Dataverse SDK#171

Open
abelmilash-msft wants to merge 52 commits into
mainfrom
users/abelmilash/async-phase2
Open

Add async support for Dataverse SDK#171
abelmilash-msft wants to merge 52 commits into
mainfrom
users/abelmilash/async-phase2

Conversation

@abelmilash-msft
Copy link
Copy Markdown
Contributor

  • Introduces the complete aio/ async package mirroring the sync SDK:
    • HTTP layer: _AsyncHttpClient wrapping aiohttp with identical retry, backoff, and timeout logic
    • Auth: _AsyncAuthManager for async Azure Identity token acquisition
    • OData client: _AsyncODataClient — full CRUD, SQL-over-API, table/column metadata, file upload, and relationship operations
    • Batch: _AsyncBatchClient with _SyncResponseWrapper bridging the async HTTP response to the shared sync multipart parser in _BatchBase
    • Operation namespaces: records, tables, query, files, batch, dataframe — all mirroring their sync counterparts
    • Top-level client: AsyncDataverseClient with lazy init, async context manager, and session lifecycle management
  • Adds pytest-asyncio (asyncio_mode = auto) and aiohttp as an optional dependency (pip install PowerPlatform-Dataverse-Client[async]).

Test plan

  • 1,758 unit tests pass (389 new async tests, ~95% coverage of the async layer)
  • black --check passes on all files

Samson Gebre and others added 19 commits May 5, 2026 20:53
…o v1

- Implemented unit tests for the `list_pages` method in `TestListPages` class, covering various scenarios including iterator return, page content validation, and parameter passing.
- Added checks for deprecation warnings to ensure no warnings are raised during the usage of `list_pages`.
- Introduced a new migration script `migrate_v0_to_v1.py` to automate the transition from beta (v0) to GA (v1) API calls, including method renaming and argument adjustments.
- Created a new `tools` directory to house the migration script.
…pdate deprecated methods

- Added simple and advanced streaming options in SKILL.md for records.list_pages() and execute_pages().
- Updated QueryBuilder to replace records.get() with records.list() in documentation and method calls.
- Improved unit tests to validate new streaming functionality and ensure correct method delegation.
- Changed method name from `fetch_xml` to `fetchxml` across the codebase for consistency.
- Updated relevant documentation to reflect the new method name.
- Added a new example script for FetchXML usage demonstrating various scenarios.
- Adjusted unit tests to accommodate the method name change and ensure proper functionality.
…ovals; modify prodev_quick_start.py for additional parameter; enhance pyproject.toml for migration tool; refine migrate_v0_to_v1.py documentation and usage instructions.
…ng and warnings; update migration tool for client variable support and manual review detection.
…agination options

- Added `include_annotations` parameter to `_RecordGet` and `_RecordList` classes for OData requests.
- Updated `_BatchClient` to handle new parameters in batch operations.
- Enhanced `_ODataClient` methods to support `include_annotations`, `expand`, `page_size`, and `count` parameters.
- Modified `BatchRecordOperations` to pass new parameters in batch record retrieval and listing methods.
- Updated `RecordOperations` to include new parameters for retrieving and listing records.
- Added unit tests to validate the new functionality for batch operations and record retrieval.
- Implemented migration tool updates to handle changes in method signatures and ensure backward compatibility.
Extract all I/O-free methods from _ODataClient and _BatchClient into
new shared base classes (_ODataBase, _BatchBase) so that a future async
sibling can inherit the same pure logic without duplicating code.

- Add data/_odata_base.py: _ODataBase with URL builders, payload
  constructors, cache helpers, _RequestContext, _USER_AGENT,
  _DEFAULT_EXPECTED_STATUSES, _extract_pagingcookie, _GUID_RE,
  _CALL_SCOPE_CORRELATION_ID, and _http_logger initialisation
- Add data/_batch_base.py: _BatchBase with all intent dataclasses,
  multipart serialisation, response parsing, and the pure table resolvers
- _ODataClient and _BatchClient now inherit from the respective base
  classes and retain only the I/O-dependent methods
- Update test patch target for urlparse to _odata_base module
- All 1223 unit tests pass; overall coverage remains at 94%

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The async batch client will pass an async OData client that inherits
from _ODataBase, not _ODataClient. _BatchBase only calls pure _build_*
methods defined on _ODataBase, so the broader base type is correct.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Add _ODataBase.close() to own cache clearing and logger teardown;
  _ODataClient.close() now delegates via super() then closes _http only
- Reorder _ODataClient bases to (_FileUploadMixin, _RelationshipOperationsMixin, _ODataBase)
  so mixins are searched before the base, matching Python MRO convention

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…base

Helpers moved to _BatchBase (_CRLF, _BOUNDARY_RE, _extract_boundary, etc.)
are no longer re-imported by _BatchClient. Two test files updated to import
these helpers directly from _batch_base where they now live.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
… missing imports to _ODataBase

- Remove _SQL_* patterns and _sql_guardrails from _ODataClient (now inherited from _ODataBase)
- Add warnings import to _odata_base.py (needed by _sql_guardrails)
- Add VALIDATION_SQL_WRITE_BLOCKED and VALIDATION_SQL_UNSUPPORTED_SYNTAX imports to _odata_base.py
- Add missing table name lowercasing logic to _ODataBase._build_lookup_field_models
@abelmilash-msft abelmilash-msft force-pushed the users/abelmilash/async-phase2 branch from e491241 to b848324 Compare May 11, 2026 23:53
@abelmilash-msft abelmilash-msft marked this pull request as ready for review May 12, 2026 06:29
@abelmilash-msft abelmilash-msft requested a review from a team as a code owner May 12, 2026 06:29
Copilot AI review requested due to automatic review settings May 12, 2026 06:29
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot wasn't able to review this pull request because it exceeds the maximum number of lines (20,000). Try reducing the number of changed lines and requesting a review from Copilot again.

Abel Milash and others added 2 commits May 14, 2026 10:15
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Add _BatchContext Protocol to _batch_base.py; re-type BatchRecordOperations,
  BatchTableOperations, BatchQueryOperations, BatchDataFrameOperations __init__
  from BatchRequest to _BatchContext — removes type: ignore on AsyncBatchRequest
- Extract _QueryBuilderBase from QueryBuilder with all fluent methods and build();
  QueryBuilder inherits base and keeps execute/execute_pages/to_dataframe;
  AsyncQueryBuilder will inherit base directly, eliminating deprecated sync surface

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@abelmilash-msft abelmilash-msft force-pushed the users/abelmilash/async-phase2 branch from b32987a to ad9297e Compare May 14, 2026 17:28
Comment thread .azdo/ci-pr.yaml
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Warning

ADO PR pipeline YAML change detected

This PR modifies .azdo/ci-pr.yaml. After merge, Azure DevOps may disable or require approval for the PR validation pipeline.

Action required (post-merge): Re-enable / approve the updated YAML for:

Please resolve this comment after completing the post-merge steps.

Comment thread .azdo/ci-pr.yaml
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Warning

ADO PR pipeline YAML change detected

This PR modifies .azdo/ci-pr.yaml. After merge, Azure DevOps may disable or require approval for the PR validation pipeline.

Action required (post-merge): Re-enable / approve the updated YAML for:

Please resolve this comment after completing the post-merge steps.

Comment thread .azdo/ci-pr.yaml
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Warning

ADO PR pipeline YAML change detected

This PR modifies .azdo/ci-pr.yaml. After merge, Azure DevOps may disable or require approval for the PR validation pipeline.

Action required (post-merge): Re-enable / approve the updated YAML for:

Please resolve this comment after completing the post-merge steps.

Comment thread .azdo/ci-pr.yaml
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Warning

ADO PR pipeline YAML change detected

This PR modifies .azdo/ci-pr.yaml. After merge, Azure DevOps may disable or require approval for the PR validation pipeline.

Action required (post-merge): Re-enable / approve the updated YAML for:

Please resolve this comment after completing the post-merge steps.

Comment thread .azdo/ci-pr.yaml
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Warning

ADO PR pipeline YAML change detected

This PR modifies .azdo/ci-pr.yaml. After merge, Azure DevOps may disable or require approval for the PR validation pipeline.

Action required (post-merge): Re-enable / approve the updated YAML for:

Please resolve this comment after completing the post-merge steps.

Abel Milash and others added 24 commits May 14, 2026 22:44
Self (from typing, under TYPE_CHECKING) is the idiomatic replacement for
the self-referential TypeVar pattern. With `from __future__ import annotations`
already in place, Self is never evaluated at runtime — only used by type
checkers. Method signatures in docs now show `Self` instead of the private
`_QB` TypeVar.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Introduces AsyncDataverseClient and the complete aio/ async stack mirroring
the sync client: _AsyncODataClient (CRUD, SQL, metadata, file upload,
relationships), _AsyncBatchClient with _SyncResponseWrapper bridge,
_AsyncHttpClient with aiohttp and identical retry/timeout logic, and all
async operation namespaces (records, tables, query, files, batch, dataframe).

Fixes a cold-start race in _bulk_fetch_picklists by adding asyncio.Lock
(double-checked locking pattern) -- concurrent coroutines no longer issue
redundant metadata fetches for the same table.

Adds pytest-asyncio (asyncio_mode=auto) and aiohttp optional dependency.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
AsyncDataverseClient is imported from the module directly:
from PowerPlatform.Dataverse.aio.async_client import AsyncDataverseClient

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
AsyncBatchRecordOperations, AsyncBatchTableOperations,
AsyncBatchQueryOperations, AsyncBatchDataFrameOperations, and
AsyncChangeSetRecordOperations were byte-for-byte copies of their sync
counterparts with no async methods or I/O. They are removed and the sync
classes from operations.batch are imported and used directly.

AsyncChangeSet is kept — it needs __aenter__/__aexit__ for
`async with batch.changeset()`.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…ions support

- Add retrieve() and list() and list_pages() to AsyncRecordOperations (GA
  replacements for deprecated get())
- Add expand and include_annotations parameters to _AsyncODataClient._get()
  and _build_get() to match sync _ODataClient
- Add _build_list() to _AsyncODataClient for batch list support
- Add _RecordList handling to _AsyncBatchClient._resolve_item() and
  _resolve_record_list(); fix _resolve_record_get to pass expand and
  include_annotations
- Update test for _resolve_record_get to match new call signature

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…dList

- TestAsyncRecordRetrieve (11 tests): return type, all params forwarded,
  404 → None, non-404 re-raised, ValueError re-raised, no DeprecationWarning,
  record.id and record.table set correctly
- TestAsyncRecordList (15 tests): QueryResult return, multi-page collection,
  all query params forwarded, FilterExpression → str conversion,
  to_dataframe(), no DeprecationWarning
- TestAsyncRecordListPages (12 tests): async generator type, QueryResult per
  page, page contents, all params forwarded, no DeprecationWarning
- TestResolveRecordGet: 3 new tests covering expand, include_annotations,
  and combined forwarding to _build_get
- TestResolveRecordList (11 tests): dispatch, all _RecordList fields
  forwarded to _build_list
- Add _build_list AsyncMock to _make_batch_client() helper
- Add _RecordList import to test file

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…atch tests

pyproject.toml:
- Omit _skill_installer.py (CLI installer, not SDK logic) and
  extensions/__init__.py (empty placeholder) from coverage measurement

tests/unit/aio/data/test_async_batch_internal.py:
- TestSyncResponseWrapper: json() returns payload, None, status_code, text
- TestExecuteEdgeCases: batch size exceeded raises, json parse failure
  falls back to empty dict
- TestResolveAllEdgeCases: empty changeset silently skipped, non-changeset
  item resolved and extended
- TestResolveItemDispatch: one test per intent type exercising every branch
  of _resolve_item() (record update/upsert, all table types, sql, unknown)
- Add _build_list, _build_create_entity, _build_get_entity,
  _build_list_entities, _build_create_relationship, _build_delete_relationship,
  _build_get_relationship, _build_lookup_field_models to _make_batch_client()
- Add imports for _SyncResponseWrapper, _MAX_BATCH_SIZE, and all intent types

Result: 2091 tests, 96.63% coverage (was 93.64%)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…ecated surface

- Remove deprecated filter_eq/filter_ne/filter_in methods from async_records and async_query (not in GA API)
- Remove stale tests for removed methods
- Use asyncio.gather() for independent I/O: relationship sub-requests, batch delete/remove-columns, multi-record label conversion, delete_columns metadata lookups
- Fix blocking file I/O in _async_upload: asyncio.to_thread() for reads, Path instead of os.path
- Remove lazy imports in _async_relationships; move to module level
- Fix test_async_batch: wrap deprecated records.get() call in pytest.warns(DeprecationWarning)
- Remove unused Optional import from _async_batch
- Remove redundant return None statements in _async_odata

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…l() on AsyncQueryOperations

- New aio/models/async_fetchxml_query.py: async paging with cookie parsing, max-pages guard, URL-length guard
- New aio/models/async_query_builder.py: async execute()/execute_pages() over QueryBuilder fluent interface
- aio/operations/async_query.py: add builder() and fetchxml() factory methods with full validation
- tests/unit/aio/test_async_query.py: 14 new tests covering builder, fetchxml, execute, execute_pages, and all error paths

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Add examples/aio/_auth.py: AsyncInteractiveBrowserCredential wrapper that
  delegates get_token() to the sync InteractiveBrowserCredential via
  ThreadPoolExecutor, satisfying the AsyncTokenCredential protocol
- Add examples/aio/basic/: installation_example.py, functional_testing.py
- Add examples/aio/advanced/: walkthrough, batch, dataframe_operations,
  relationships, alternate_keys_upsert, fetchxml, sql_examples,
  prodev_quick_start, datascience_risk_assessment, file_upload
- Fix prodev_quick_start: create tables sequentially (Dataverse holds a
  metadata customization lock per request; concurrent creates fail);
  retry cleanup loop to handle transient SQL deadlocks

All 12 scripts validated end-to-end against a live Dataverse environment.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…coverage to 98%

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…syncBatchRequest

- AsyncQueryBuilder now inherits _QueryBuilderBase instead of QueryBuilder,
  eliminating deprecated sync execution surface (execute(by_page=...), to_dataframe())
- AsyncBatchRequest drops 4 type: ignore[arg-type] now that batch operation
  classes accept _BatchContext instead of the concrete BatchRequest

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Mirrors sync _ODataClient: reads config.operation_context in __init__ and
appends user_agent_context as a parenthesized comment to the User-Agent header.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…) content_type

- Add `context: Optional[OperationContext] = None` keyword argument to
  AsyncDataverseClient.__init__ with the same conflict validation and
  three-branch config initialization as the sync DataverseClient
- Update AsyncDataverseClient docstring to document the context parameter
- Add content_type=None to all await r.json() calls in _async_relationships.py
  for consistency with _async_odata.py

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Mirror TestOperationContextClient from test_operation_context.py:
- context= kwarg stores OperationContext in _config
- no context= leaves operation_context as None
- config= + context= together raise ValueError
- config= alone wires operation_context correctly

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Adds _HttpResponse to _async_http.py — a materialized response that
buffers the body bytes in _request, exposing sync .text and .json()
so callers need no await. Eliminates _SyncResponseWrapper from
_async_batch.py and removes all remaining await r.json() calls
throughout the async layer (_async_odata.py, _async_relationships.py).
Also removes aiohttp.ContentTypeError from except clauses now that
json() is a plain json.loads call (ValueError covers it).

All 2155 tests pass.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- async_fetchxml_query.py: fix leftover await r.json() -> r.json()
- _async_odata.py: add TYPE_CHECKING guard for aiohttp annotation
- test helpers: replace AsyncMock with MagicMock for .json/.text on
  _AsyncResponse-compatible mocks (body already materialized, no await)

All 2155 tests pass.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…ase)

Follows the same cleanup applied to the sync _ODataClient on the
refactoring branch. _operation_context is now initialized once in
_ODataBase.__init__ and inherited by both sync and async subclasses.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@abelmilash-msft abelmilash-msft force-pushed the users/abelmilash/async-phase2 branch from 500c19f to 55e3359 Compare May 15, 2026 05:44
Comment thread .azdo/ci-pr.yaml
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Warning

ADO PR pipeline YAML change detected

This PR modifies .azdo/ci-pr.yaml. After merge, Azure DevOps may disable or require approval for the PR validation pipeline.

Action required (post-merge): Re-enable / approve the updated YAML for:

Please resolve this comment after completing the post-merge steps.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Comment thread .azdo/ci-pr.yaml
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Warning

ADO PR pipeline YAML change detected

This PR modifies .azdo/ci-pr.yaml. After merge, Azure DevOps may disable or require approval for the PR validation pipeline.

Action required (post-merge): Re-enable / approve the updated YAML for:

Please resolve this comment after completing the post-merge steps.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants